agora inbox for [email protected]help / color / mirror / Atom feed
Re: Patch: add timing of buffer I/O requests 12552+ messages / 4 participants [nested] [flat]
* Re: Patch: add timing of buffer I/O requests @ 2012-03-21 15:01 Robert Haas <[email protected]> 2012-03-22 23:38 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> 0 siblings, 1 reply; 12552+ messages in thread From: Robert Haas @ 2012-03-21 15:01 UTC (permalink / raw) To: Ants Aasma <[email protected]>; +Cc: Greg Smith <[email protected]>; pgsql-hackers On Fri, Feb 24, 2012 at 2:23 PM, Ants Aasma <[email protected]> wrote: > On Wed, Feb 22, 2012 at 6:35 PM, Ants Aasma <[email protected]> wrote: >>> Some implementation notes. This currently fails regression test >>> create_function_3, haven't looked into why yet. >> >> I'll take a look at it. > > The failure was due to leakproof changes to pgproc. Attached patches > are adjusted accordingly and rebased over Robert's blocks dirtied > patch. This seems to have bitrotted again. :-( Can you please rebase again? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 12552+ messages in thread
* Re: Patch: add timing of buffer I/O requests 2012-03-21 15:01 Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> @ 2012-03-22 23:38 ` Ants Aasma <[email protected]> 2012-03-27 18:58 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 0 siblings, 1 reply; 12552+ messages in thread From: Ants Aasma @ 2012-03-22 23:38 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Greg Smith <[email protected]>; pgsql-hackers On Wed, Mar 21, 2012 at 5:01 PM, Robert Haas <[email protected]> wrote: > This seems to have bitrotted again. :-( > > Can you please rebase again? Rebased. Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de Attachments: [text/x-patch] io-stats.v5.patch (49.1K, ../../CA+CSw_tfw+Vcgy1WXVx5kt5=739D0_aTXxzh5Ubf-SABDASYqg@mail.gmail.com/2-io-stats.v5.patch) download | inline diff: diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index 3e17875..97e8e51 100644 --- a/doc/src/sgml/config.sgml +++ b/doc/src/sgml/config.sgml @@ -4284,6 +4284,25 @@ COPY postgres_log FROM '/full/path/to/logfile.csv' WITH csv; </listitem> </varlistentry> + <varlistentry id="guc-track-iotiming" xreflabel="track_iotiming"> + <term><varname>track_iotiming</varname> (<type>boolean</type>)</term> + <indexterm> + <primary><varname>track_iotiming</> configuration parameter</primary> + </indexterm> + <listitem> + <para> + Enables timing of database I/O calls. + This parameter is off by default, because it may cause significant + overhead if the platform doesn't support fast timing information. + Only superusers can change this setting. + </para> + <para> + You can use the <xref linkend="pgtesttiming"> tool to find out the + overhead of timing on your system. + </para> + </listitem> + </varlistentry> + <varlistentry id="guc-track-functions" xreflabel="track_functions"> <term><varname>track_functions</varname> (<type>enum</type>)</term> <indexterm> diff --git a/doc/src/sgml/monitoring.sgml b/doc/src/sgml/monitoring.sgml index 840e54a..ba69382 100644 --- a/doc/src/sgml/monitoring.sgml +++ b/doc/src/sgml/monitoring.sgml @@ -116,7 +116,8 @@ postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re <productname>PostgreSQL</productname>'s <firstterm>statistics collector</> is a subsystem that supports collection and reporting of information about server activity. Presently, the collector can count accesses to tables - and indexes in both disk-block and individual-row terms. It also tracks + and indexes in both disk-block and individual-row terms and time disk-block + accesses. It also tracks the total number of rows in each table, and information about vacuum and analyze actions for each table. It can also count calls to user-defined functions and the total time spent in each one. @@ -145,6 +146,11 @@ postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re </para> <para> + The parameter <xref linkend="guc-track-iotiming"> enables timing of I/O + requests. + </para> + + <para> The parameter <xref linkend="guc-track-functions"> enables tracking of usage of user-defined functions. </para> @@ -483,7 +489,10 @@ postgres: <replaceable>user</> <replaceable>database</> <replaceable>host</> <re handles disk I/O, data that is not in the <productname>PostgreSQL</> buffer cache might still reside in the kernel's I/O cache, and might therefore still be fetched without - requiring a physical read. Users interested in obtaining more + requiring a physical read. + Timing info shows how much user queries were delayed by buffer reads + in aggregate. + Users interested in obtaining more detailed information on <productname>PostgreSQL</> I/O behavior are advised to use the <productname>PostgreSQL</> statistics collector in combination with operating system utilities that allow insight @@ -885,6 +894,14 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, the <function>pg_stat_get_db_blocks_hit</function> function.</entry> </row> <row> + <entry>blks_time</entry> + <entry><type>bigint</></entry> + <entry>Total time spent waiting for disk blocks to be read + in this database, in microseconds. + This value can also be returned by directly calling + the <function>pg_stat_get_db_blocks_time</function> function.</entry> + </row> + <row> <entry>tup_returned</entry> <entry><type>bigint</></entry> <entry>The number of rows returned by queries in this database. @@ -1441,6 +1458,14 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, the <function>pg_stat_get_blocks_hit</function> function.</entry> </row> <row> + <entry>heap_blks_time</entry> + <entry><type>bigint</></entry> + <entry>Total time spent waiting for disk blocks to be read + from this table, in microseconds. + This value can also be returned by directly calling + the <function>pg_stat_get_blocks_time</function> function.</entry> + </row> + <row> <entry>idx_blks_read</entry> <entry><type>name</></entry> <entry>Number of disk blocks read from all indexes on this table</entry> @@ -1451,6 +1476,12 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <entry>Number of buffer hits in all indexes of this table.</entry> </row> <row> + <entry>idx_blks_time</entry> + <entry><type>bigint</></entry> + <entry>Total time spent waiting for disk blocks to be read + from all indexes on this table, in microseconds.</entry> + </row> + <row> <entry>toast_blks_read</entry> <entry><type>name</></entry> <entry>Number of disk blocks read from this table's TOAST table (if any)</entry> @@ -1461,6 +1492,12 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <entry>Number of buffer hits in this table's TOAST table (if any)</entry> </row> <row> + <entry>toast_blks_time</entry> + <entry><type>bigint</></entry> + <entry>Total time spent waiting for disk blocks to be read + from this table's TOAST table (if any), in microseconds.</entry> + </row> + <row> <entry>tidx_blks_read</entry> <entry><type>name</></entry> <entry>Number of disk blocks read from this table's TOAST table index (if any)</entry> @@ -1470,6 +1507,12 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, <entry><type>name</></entry> <entry>Number of buffer hits in this table's TOAST table index (if any)</entry> </row> + <row> + <entry>tidx_blks_time</entry> + <entry><type>bigint</></entry> + <entry>Total time spent waiting for disk blocks to be read + from this table's TOAST table index (if any), in microseconds.</entry> + </row> </tbody> </tgroup> </table> @@ -1536,6 +1579,14 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, This value can also be returned by directly calling the <function>pg_stat_get_blocks_hit</function> function.</entry> </row> + <row> + <entry>idx_blks_time</entry> + <entry><type>bigint</></entry> + <entry>Total time spent waiting for disk blocks to be read + from the index, in microseconds. + This value can also be returned by directly calling + the <function>pg_stat_get_blocks_time</function> function.</entry> + </row> </tbody> </tgroup> </table> @@ -1703,6 +1754,14 @@ SELECT pg_stat_get_backend_pid(s.backendid) AS pid, </entry> </row> + <row> + <entry><literal><function>pg_stat_get_xact_blocks_time</function>(<type>oid</type>)</literal></entry> + <entry><type>bigint</type></entry> + <entry> + Total time spent waiting for disk block requests for table or index, in the current transaction, + in microseconds. + </entry> + </row> <row> <entry><literal><function>pg_stat_get_wal_senders()</function></literal></entry> diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql index ab594eb..430bea1 100644 --- a/src/backend/catalog/system_views.sql +++ b/src/backend/catalog/system_views.sql @@ -440,15 +440,19 @@ CREATE VIEW pg_statio_all_tables AS pg_stat_get_blocks_fetched(C.oid) - pg_stat_get_blocks_hit(C.oid) AS heap_blks_read, pg_stat_get_blocks_hit(C.oid) AS heap_blks_hit, + pg_stat_get_blocks_time(C.oid) AS heap_blks_time, sum(pg_stat_get_blocks_fetched(I.indexrelid) - pg_stat_get_blocks_hit(I.indexrelid))::bigint AS idx_blks_read, sum(pg_stat_get_blocks_hit(I.indexrelid))::bigint AS idx_blks_hit, + sum(pg_stat_get_blocks_time(I.indexrelid))::bigint AS idx_blks_time, pg_stat_get_blocks_fetched(T.oid) - pg_stat_get_blocks_hit(T.oid) AS toast_blks_read, pg_stat_get_blocks_hit(T.oid) AS toast_blks_hit, + pg_stat_get_blocks_time(T.oid) AS toast_blks_time, pg_stat_get_blocks_fetched(X.oid) - pg_stat_get_blocks_hit(X.oid) AS tidx_blks_read, - pg_stat_get_blocks_hit(X.oid) AS tidx_blks_hit + pg_stat_get_blocks_hit(X.oid) AS tidx_blks_hit, + pg_stat_get_blocks_time(X.oid) AS tidx_blks_time FROM pg_class C LEFT JOIN pg_index I ON C.oid = I.indrelid LEFT JOIN pg_class T ON C.reltoastrelid = T.oid LEFT JOIN @@ -502,7 +506,8 @@ CREATE VIEW pg_statio_all_indexes AS I.relname AS indexrelname, pg_stat_get_blocks_fetched(I.oid) - pg_stat_get_blocks_hit(I.oid) AS idx_blks_read, - pg_stat_get_blocks_hit(I.oid) AS idx_blks_hit + pg_stat_get_blocks_hit(I.oid) AS idx_blks_hit, + pg_stat_get_blocks_time(I.oid) AS idx_blks_time FROM pg_class C JOIN pg_index X ON C.oid = X.indrelid JOIN pg_class I ON I.oid = X.indexrelid @@ -526,7 +531,8 @@ CREATE VIEW pg_statio_all_sequences AS C.relname AS relname, pg_stat_get_blocks_fetched(C.oid) - pg_stat_get_blocks_hit(C.oid) AS blks_read, - pg_stat_get_blocks_hit(C.oid) AS blks_hit + pg_stat_get_blocks_hit(C.oid) AS blks_hit, + pg_stat_get_blocks_time(C.oid) AS blks_time FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace) WHERE C.relkind = 'S'; @@ -595,6 +601,7 @@ CREATE VIEW pg_stat_database AS pg_stat_get_db_blocks_fetched(D.oid) - pg_stat_get_db_blocks_hit(D.oid) AS blks_read, pg_stat_get_db_blocks_hit(D.oid) AS blks_hit, + pg_stat_get_db_blocks_time(D.oid) AS blks_time, pg_stat_get_db_tuples_returned(D.oid) AS tup_returned, pg_stat_get_db_tuples_fetched(D.oid) AS tup_fetched, pg_stat_get_db_tuples_inserted(D.oid) AS tup_inserted, diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index a14cae1..3707f7e 100644 --- a/src/backend/commands/explain.c +++ b/src/backend/commands/explain.c @@ -1236,9 +1236,11 @@ ExplainNode(PlanState *planstate, List *ancestors, usage->local_blks_written > 0); bool has_temp = (usage->temp_blks_read > 0 || usage->temp_blks_written > 0); + bool has_timing = (!INSTR_TIME_IS_ZERO(usage->time_read) || + !INSTR_TIME_IS_ZERO(usage->time_write)); /* Show only positive counter values. */ - if (has_shared || has_local || has_temp) + if (has_shared || has_local || has_temp || has_timing) { appendStringInfoSpaces(es->str, es->indent * 2); appendStringInfoString(es->str, "Buffers:"); @@ -1289,6 +1291,18 @@ ExplainNode(PlanState *planstate, List *ancestors, appendStringInfo(es->str, " written=%ld", usage->temp_blks_written); } + if (has_timing) + { + appendStringInfoString(es->str, " timing"); + if (!INSTR_TIME_IS_ZERO(usage->time_read)) { + appendStringInfo(es->str, " read=%0.2f", + INSTR_TIME_GET_MILLISEC(usage->time_read)); + } + if (!INSTR_TIME_IS_ZERO(usage->time_write)) { + appendStringInfo(es->str, " write=%0.2f", + INSTR_TIME_GET_MILLISEC(usage->time_write)); + } + } appendStringInfoChar(es->str, '\n'); } } @@ -1304,6 +1318,8 @@ ExplainNode(PlanState *planstate, List *ancestors, ExplainPropertyLong("Local Written Blocks", usage->local_blks_written, es); ExplainPropertyLong("Temp Read Blocks", usage->temp_blks_read, es); ExplainPropertyLong("Temp Written Blocks", usage->temp_blks_written, es); + ExplainPropertyFloat("Read Waits", INSTR_TIME_GET_MILLISEC(usage->time_read), 3, es); + ExplainPropertyFloat("Write Waits", INSTR_TIME_GET_MILLISEC(usage->time_write), 3, es); } } diff --git a/src/backend/executor/instrument.c b/src/backend/executor/instrument.c index 6e9f450..92e56d1 100644 --- a/src/backend/executor/instrument.c +++ b/src/backend/executor/instrument.c @@ -145,4 +145,6 @@ BufferUsageAccumDiff(BufferUsage *dst, dst->local_blks_written += add->local_blks_written - sub->local_blks_written; dst->temp_blks_read += add->temp_blks_read - sub->temp_blks_read; dst->temp_blks_written += add->temp_blks_written - sub->temp_blks_written; + INSTR_TIME_ACCUM_DIFF(dst->time_read, add->time_read, sub->time_read); + INSTR_TIME_ACCUM_DIFF(dst->time_write, add->time_write, sub->time_write); } diff --git a/src/backend/postmaster/pgstat.c b/src/backend/postmaster/pgstat.c index a53fc52..e9f6a3b 100644 --- a/src/backend/postmaster/pgstat.c +++ b/src/backend/postmaster/pgstat.c @@ -3338,6 +3338,7 @@ pgstat_get_db_entry(Oid databaseid, bool create) result->n_xact_rollback = 0; result->n_blocks_fetched = 0; result->n_blocks_hit = 0; + INSTR_TIME_SET_ZERO(result->blocks_time); result->n_tuples_returned = 0; result->n_tuples_fetched = 0; result->n_tuples_inserted = 0; @@ -3412,6 +3413,7 @@ pgstat_get_tab_entry(PgStat_StatDBEntry *dbentry, Oid tableoid, bool create) result->changes_since_analyze = 0; result->blocks_fetched = 0; result->blocks_hit = 0; + INSTR_TIME_SET_ZERO(result->blocks_time); result->vacuum_timestamp = 0; result->vacuum_count = 0; result->autovac_vacuum_timestamp = 0; @@ -4100,6 +4102,7 @@ pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len) tabentry->changes_since_analyze = tabmsg->t_counts.t_changed_tuples; tabentry->blocks_fetched = tabmsg->t_counts.t_blocks_fetched; tabentry->blocks_hit = tabmsg->t_counts.t_blocks_hit; + tabentry->blocks_time = tabmsg->t_counts.t_blocks_time; tabentry->vacuum_timestamp = 0; tabentry->vacuum_count = 0; @@ -4127,6 +4130,7 @@ pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len) tabentry->changes_since_analyze += tabmsg->t_counts.t_changed_tuples; tabentry->blocks_fetched += tabmsg->t_counts.t_blocks_fetched; tabentry->blocks_hit += tabmsg->t_counts.t_blocks_hit; + INSTR_TIME_ADD(tabentry->blocks_time, tabmsg->t_counts.t_blocks_time); } /* Clamp n_live_tuples in case of negative delta_live_tuples */ @@ -4144,6 +4148,7 @@ pgstat_recv_tabstat(PgStat_MsgTabstat *msg, int len) dbentry->n_tuples_deleted += tabmsg->t_counts.t_tuples_deleted; dbentry->n_blocks_fetched += tabmsg->t_counts.t_blocks_fetched; dbentry->n_blocks_hit += tabmsg->t_counts.t_blocks_hit; + INSTR_TIME_ADD(dbentry->blocks_time, tabmsg->t_counts.t_blocks_time); } } @@ -4257,6 +4262,7 @@ pgstat_recv_resetcounter(PgStat_MsgResetcounter *msg, int len) dbentry->n_xact_rollback = 0; dbentry->n_blocks_fetched = 0; dbentry->n_blocks_hit = 0; + INSTR_TIME_SET_ZERO(dbentry->blocks_time); dbentry->n_tuples_returned = 0; dbentry->n_tuples_fetched = 0; dbentry->n_tuples_inserted = 0; diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c index 3924a51..a28d8e3 100644 --- a/src/backend/storage/buffer/bufmgr.c +++ b/src/backend/storage/buffer/bufmgr.c @@ -67,6 +67,7 @@ bool zero_damaged_pages = false; int bgwriter_lru_maxpages = 100; double bgwriter_lru_multiplier = 2.0; +bool track_iotiming = false; /* * How many buffers PrefetchBuffer callers should try to stay ahead of their @@ -86,7 +87,7 @@ static volatile BufferDesc *PinCountWaitBuf = NULL; static Buffer ReadBuffer_common(SMgrRelation reln, char relpersistence, ForkNumber forkNum, BlockNumber blockNum, ReadBufferMode mode, BufferAccessStrategy strategy, - bool *hit); + bool *hit, instr_time *io_time); static bool PinBuffer(volatile BufferDesc *buf, BufferAccessStrategy strategy); static void PinBuffer_Locked(volatile BufferDesc *buf); static void UnpinBuffer(volatile BufferDesc *buf, bool fixOwner); @@ -224,6 +225,7 @@ ReadBufferExtended(Relation reln, ForkNumber forkNum, BlockNumber blockNum, ReadBufferMode mode, BufferAccessStrategy strategy) { bool hit; + instr_time io_time; Buffer buf; /* Open it at the smgr level if not already done */ @@ -245,9 +247,11 @@ ReadBufferExtended(Relation reln, ForkNumber forkNum, BlockNumber blockNum, */ pgstat_count_buffer_read(reln); buf = ReadBuffer_common(reln->rd_smgr, reln->rd_rel->relpersistence, - forkNum, blockNum, mode, strategy, &hit); + forkNum, blockNum, mode, strategy, &hit, &io_time); if (hit) pgstat_count_buffer_hit(reln); + else + pgstat_count_buffer_time(reln, io_time); return buf; } @@ -267,11 +271,12 @@ ReadBufferWithoutRelcache(RelFileNode rnode, ForkNumber forkNum, BufferAccessStrategy strategy) { bool hit; + instr_time io_time; SMgrRelation smgr = smgropen(rnode, InvalidBackendId); return ReadBuffer_common(smgr, RELPERSISTENCE_PERMANENT, forkNum, blockNum, - mode, strategy, &hit); + mode, strategy, &hit, &io_time); } @@ -279,19 +284,22 @@ ReadBufferWithoutRelcache(RelFileNode rnode, ForkNumber forkNum, * ReadBuffer_common -- common logic for all ReadBuffer variants * * *hit is set to true if the request was satisfied from shared buffer cache. + * If track_iotiming is enabled, *io_time is set to the time the read took. */ static Buffer ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, BlockNumber blockNum, ReadBufferMode mode, - BufferAccessStrategy strategy, bool *hit) + BufferAccessStrategy strategy, bool *hit, instr_time *io_time) { volatile BufferDesc *bufHdr; Block bufBlock; bool found; bool isExtend; bool isLocalBuf = SmgrIsTemp(smgr); + instr_time io_start, io_end; *hit = false; + INSTR_TIME_SET_ZERO(*io_time); /* Make sure we will have room to remember the buffer pin */ ResourceOwnerEnlargeBuffers(CurrentResourceOwner); @@ -437,8 +445,18 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum, MemSet((char *) bufBlock, 0, BLCKSZ); else { + if (track_iotiming) + INSTR_TIME_SET_CURRENT(io_start); + smgrread(smgr, forkNum, blockNum, (char *) bufBlock); + if (track_iotiming) + { + INSTR_TIME_SET_CURRENT(io_end); + INSTR_TIME_ACCUM_DIFF(*io_time, io_end, io_start); + INSTR_TIME_ADD(pgBufferUsage.time_read, *io_time); + } + /* check for garbage data */ if (!PageHeaderIsValid((PageHeader) bufBlock)) { @@ -1874,6 +1892,7 @@ FlushBuffer(volatile BufferDesc *buf, SMgrRelation reln) { XLogRecPtr recptr; ErrorContextCallback errcontext; + instr_time io_start, io_end; /* * Acquire the buffer's io_in_progress lock. If StartBufferIO returns @@ -1921,12 +1940,21 @@ FlushBuffer(volatile BufferDesc *buf, SMgrRelation reln) buf->flags &= ~BM_JUST_DIRTIED; UnlockBufHdr(buf); + if (track_iotiming) + INSTR_TIME_SET_CURRENT(io_start); + smgrwrite(reln, buf->tag.forkNum, buf->tag.blockNum, (char *) BufHdrGetBlock(buf), false); + if (track_iotiming) + { + INSTR_TIME_SET_CURRENT(io_end); + INSTR_TIME_ACCUM_DIFF(pgBufferUsage.time_write, io_end, io_start); + } + pgBufferUsage.shared_blks_written++; /* @@ -2670,6 +2698,7 @@ WaitIO(volatile BufferDesc *buf) static bool StartBufferIO(volatile BufferDesc *buf, bool forInput) { + instr_time wait_start, wait_end; Assert(!InProgressBuf); for (;;) @@ -2678,7 +2707,28 @@ StartBufferIO(volatile BufferDesc *buf, bool forInput) * Grab the io_in_progress lock so that other processes can wait for * me to finish the I/O. */ - LWLockAcquire(buf->io_in_progress_lock, LW_EXCLUSIVE); + if (forInput && track_iotiming) { + /* + * We need to time the lock wait to account for I/O waits where + * someone else is doing the work for us. Conditional acquire + * avoids double timing overhead when we do the I/O ourselves. + */ + if (!LWLockConditionalAcquire(buf->io_in_progress_lock, LW_EXCLUSIVE)) { + INSTR_TIME_SET_CURRENT(wait_start); + + LWLockAcquire(buf->io_in_progress_lock, LW_EXCLUSIVE); + + /* + * Only do backend local accounting, stats collector will get the + * wait from the backend doing the I/O. + */ + INSTR_TIME_SET_CURRENT(wait_end); + INSTR_TIME_ACCUM_DIFF(pgBufferUsage.time_read, wait_end, wait_start); + } + } else { + LWLockAcquire(buf->io_in_progress_lock, LW_EXCLUSIVE); + } + LockBufHdr(buf); diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 68b2527..d0b6c57 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -22,6 +22,7 @@ #include "utils/builtins.h" #include "utils/inet.h" #include "utils/timestamp.h" +#include "portability/instr_time.h" /* bogus ... these externs should be in a header file */ extern Datum pg_stat_get_numscans(PG_FUNCTION_ARGS); @@ -35,6 +36,7 @@ extern Datum pg_stat_get_live_tuples(PG_FUNCTION_ARGS); extern Datum pg_stat_get_dead_tuples(PG_FUNCTION_ARGS); extern Datum pg_stat_get_blocks_fetched(PG_FUNCTION_ARGS); extern Datum pg_stat_get_blocks_hit(PG_FUNCTION_ARGS); +extern Datum pg_stat_get_blocks_time(PG_FUNCTION_ARGS); extern Datum pg_stat_get_last_vacuum_time(PG_FUNCTION_ARGS); extern Datum pg_stat_get_last_autovacuum_time(PG_FUNCTION_ARGS); extern Datum pg_stat_get_last_analyze_time(PG_FUNCTION_ARGS); @@ -67,6 +69,7 @@ extern Datum pg_stat_get_db_xact_commit(PG_FUNCTION_ARGS); extern Datum pg_stat_get_db_xact_rollback(PG_FUNCTION_ARGS); extern Datum pg_stat_get_db_blocks_fetched(PG_FUNCTION_ARGS); extern Datum pg_stat_get_db_blocks_hit(PG_FUNCTION_ARGS); +extern Datum pg_stat_get_db_blocks_time(PG_FUNCTION_ARGS); extern Datum pg_stat_get_db_tuples_returned(PG_FUNCTION_ARGS); extern Datum pg_stat_get_db_tuples_fetched(PG_FUNCTION_ARGS); extern Datum pg_stat_get_db_tuples_inserted(PG_FUNCTION_ARGS); @@ -102,6 +105,7 @@ extern Datum pg_stat_get_xact_tuples_deleted(PG_FUNCTION_ARGS); extern Datum pg_stat_get_xact_tuples_hot_updated(PG_FUNCTION_ARGS); extern Datum pg_stat_get_xact_blocks_fetched(PG_FUNCTION_ARGS); extern Datum pg_stat_get_xact_blocks_hit(PG_FUNCTION_ARGS); +extern Datum pg_stat_get_xact_blocks_time(PG_FUNCTION_ARGS); extern Datum pg_stat_get_xact_function_calls(PG_FUNCTION_ARGS); extern Datum pg_stat_get_xact_function_time(PG_FUNCTION_ARGS); @@ -292,6 +296,22 @@ pg_stat_get_blocks_hit(PG_FUNCTION_ARGS) } Datum +pg_stat_get_blocks_time(PG_FUNCTION_ARGS) +{ + Oid relid = PG_GETARG_OID(0); + double result; + PgStat_StatTabEntry *tabentry; + + if ((tabentry = pgstat_fetch_stat_tabentry(relid)) == NULL) + result = 0; + else + /* Cast overflows in about 300'000 years of io time */ + result = (int64) INSTR_TIME_GET_MICROSEC(tabentry->blocks_time); + + PG_RETURN_INT64(result); +} + +Datum pg_stat_get_last_vacuum_time(PG_FUNCTION_ARGS) { Oid relid = PG_GETARG_OID(0); @@ -1120,6 +1140,22 @@ pg_stat_get_db_blocks_hit(PG_FUNCTION_ARGS) Datum +pg_stat_get_db_blocks_time(PG_FUNCTION_ARGS) +{ + Oid dbid = PG_GETARG_OID(0); + int64 result; + PgStat_StatDBEntry *dbentry; + + if ((dbentry = pgstat_fetch_stat_dbentry(dbid)) == NULL) + result = 0; + else + result = (int64) INSTR_TIME_GET_MICROSEC(dbentry->blocks_time); + + PG_RETURN_INT64(result); +} + + +Datum pg_stat_get_db_tuples_returned(PG_FUNCTION_ARGS) { Oid dbid = PG_GETARG_OID(0); @@ -1565,6 +1601,21 @@ pg_stat_get_xact_blocks_hit(PG_FUNCTION_ARGS) } Datum +pg_stat_get_xact_blocks_time(PG_FUNCTION_ARGS) +{ + Oid relid = PG_GETARG_OID(0); + int64 result; + PgStat_TableStatus *tabentry; + + if ((tabentry = find_tabstat_entry(relid)) == NULL) + result = 0; + else + result = (int64) INSTR_TIME_GET_MICROSEC(tabentry->t_counts.t_blocks_time); + + PG_RETURN_INT64(result); +} + +Datum pg_stat_get_xact_function_calls(PG_FUNCTION_ARGS) { Oid funcid = PG_GETARG_OID(0); diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 3d2fe3e..8ea391a 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -1018,6 +1018,15 @@ static struct config_bool ConfigureNamesBool[] = true, NULL, NULL, NULL }, + { + {"track_iotiming", PGC_SUSET, STATS_COLLECTOR, + gettext_noop("Collects timing information for database IO activity."), + NULL + }, + &track_iotiming, + false, + NULL, NULL, NULL + }, { {"update_process_title", PGC_SUSET, STATS_COLLECTOR, diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index 96da086..cbae3bf 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -424,6 +424,7 @@ #track_activities = on #track_counts = on +#track_iotiming = off #track_functions = none # none, pl, all #track_activity_query_size = 1024 # (change requires restart) #update_process_title = on diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.h index 2db8489..c4ac58f 100644 --- a/src/include/catalog/pg_proc.h +++ b/src/include/catalog/pg_proc.h @@ -2680,6 +2680,12 @@ DATA(insert OID = 3063 ( pg_stat_get_buf_fsync_backend PGNSP PGUID 12 1 0 0 0 f DESCR("statistics: number of backend buffer writes that did their own fsync"); DATA(insert OID = 2859 ( pg_stat_get_buf_alloc PGNSP PGUID 12 1 0 0 0 f f f f t f s 0 0 20 "" _null_ _null_ _null_ _null_ pg_stat_get_buf_alloc _null_ _null_ _null_ )); DESCR("statistics: number of buffer allocations"); +DATA(insert OID = 3160 ( pg_stat_get_blocks_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_blocks_time _null_ _null_ _null_ )); +DESCR("statistics: duration of block reads"); +DATA(insert OID = 3161 ( pg_stat_get_db_blocks_time PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_db_blocks_time _null_ _null_ _null_ )); +DESCR("statistics: duration of block reads for database"); +DATA(insert OID = 3166 ( pg_stat_get_xact_blocks_time PGNSP PGUID 12 1 0 0 0 f f f f t f v 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_xact_blocks_time _null_ _null_ _null_ )); +DESCR("statistics: duration of block reads in current transaction"); DATA(insert OID = 2978 ( pg_stat_get_function_calls PGNSP PGUID 12 1 0 0 0 f f f f t f s 1 0 20 "26" _null_ _null_ _null_ _null_ pg_stat_get_function_calls _null_ _null_ _null_ )); DESCR("statistics: number of function calls"); diff --git a/src/include/executor/instrument.h b/src/include/executor/instrument.h index 066f684..ad13914 100644 --- a/src/include/executor/instrument.h +++ b/src/include/executor/instrument.h @@ -28,6 +28,8 @@ typedef struct BufferUsage long local_blks_written; /* # of local disk blocks written */ long temp_blks_read; /* # of temp blocks read */ long temp_blks_written; /* # of temp blocks written */ + instr_time time_read; /* time spent reading */ + instr_time time_write; /* time spent writing */ } BufferUsage; /* Flag bits included in InstrAlloc's instrument_options bitmask */ diff --git a/src/include/pgstat.h b/src/include/pgstat.h index 1281bd8..175b5df 100644 --- a/src/include/pgstat.h +++ b/src/include/pgstat.h @@ -97,6 +97,7 @@ typedef struct PgStat_TableCounts PgStat_Counter t_blocks_fetched; PgStat_Counter t_blocks_hit; + instr_time t_blocks_time; } PgStat_TableCounts; /* Possible targets for resetting cluster-wide shared values */ @@ -522,6 +523,7 @@ typedef struct PgStat_StatDBEntry PgStat_Counter n_xact_rollback; PgStat_Counter n_blocks_fetched; PgStat_Counter n_blocks_hit; + instr_time blocks_time; PgStat_Counter n_tuples_returned; PgStat_Counter n_tuples_fetched; PgStat_Counter n_tuples_inserted; @@ -573,6 +575,7 @@ typedef struct PgStat_StatTabEntry PgStat_Counter blocks_fetched; PgStat_Counter blocks_hit; + instr_time blocks_time; TimestampTz vacuum_timestamp; /* user initiated vacuum */ PgStat_Counter vacuum_count; @@ -816,6 +819,13 @@ extern void pgstat_initstats(Relation rel); if ((rel)->pgstat_info != NULL) \ (rel)->pgstat_info->t_counts.t_blocks_hit++; \ } while (0) +#define pgstat_count_buffer_time(rel, io) \ + do { \ + if ((rel)->pgstat_info != NULL) \ + INSTR_TIME_ADD( \ + (rel)->pgstat_info->t_counts.t_blocks_time, \ + (io)); \ + } while (0) extern void pgstat_count_heap_insert(Relation rel, int n); extern void pgstat_count_heap_update(Relation rel, bool hot); diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h index de1bbd0..d72bad9 100644 --- a/src/include/storage/bufmgr.h +++ b/src/include/storage/bufmgr.h @@ -48,6 +48,7 @@ extern PGDLLIMPORT int NBuffers; extern bool zero_damaged_pages; extern int bgwriter_lru_maxpages; extern double bgwriter_lru_multiplier; +extern bool track_iotiming; extern int target_prefetch_pages; /* in buf_init.c */ diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out index aaf0cca..ce6228c 100644 --- a/src/test/regress/expected/rules.out +++ b/src/test/regress/expected/rules.out @@ -1296,7 +1296,7 @@ SELECT viewname, definition FROM pg_views WHERE schemaname <> 'information_schem pg_stat_all_indexes | SELECT c.oid AS relid, i.oid AS indexrelid, n.nspname AS schemaname, c.relname, i.relname AS indexrelname, pg_stat_get_numscans(i.oid) AS idx_scan, pg_stat_get_tuples_returned(i.oid) AS idx_tup_read, pg_stat_get_tuples_fetched(i.oid) AS idx_tup_fetch FROM (((pg_class c JOIN pg_index x ON ((c.oid = x.indrelid))) JOIN pg_class i ON ((i.oid = x.indexrelid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE (c.relkind = ANY (ARRAY['r'::"char", 't'::"char"])); pg_stat_all_tables | SELECT c.oid AS relid, n.nspname AS schemaname, c.relname, pg_stat_get_numscans(c.oid) AS seq_scan, pg_stat_get_tuples_returned(c.oid) AS seq_tup_read, (sum(pg_stat_get_numscans(i.indexrelid)))::bigint AS idx_scan, ((sum(pg_stat_get_tuples_fetched(i.indexrelid)))::bigint + pg_stat_get_tuples_fetched(c.oid)) AS idx_tup_fetch, pg_stat_get_tuples_inserted(c.oid) AS n_tup_ins, pg_stat_get_tuples_updated(c.oid) AS n_tup_upd, pg_stat_get_tuples_deleted(c.oid) AS n_tup_del, pg_stat_get_tuples_hot_updated(c.oid) AS n_tup_hot_upd, pg_stat_get_live_tuples(c.oid) AS n_live_tup, pg_stat_get_dead_tuples(c.oid) AS n_dead_tup, pg_stat_get_last_vacuum_time(c.oid) AS last_vacuum, pg_stat_get_last_autovacuum_time(c.oid) AS last_autovacuum, pg_stat_get_last_analyze_time(c.oid) AS last_analyze, pg_stat_get_last_autoanalyze_time(c.oid) AS last_autoanalyze, pg_stat_get_vacuum_count(c.oid) AS vacuum_count, pg_stat_get_autovacuum_count(c.oid) AS autovacuum_count, pg_stat_get_analyze_count(c.oid) AS analyze_count, pg_stat_get_autoanalyze_count(c.oid) AS autoanalyze_count FROM ((pg_class c LEFT JOIN pg_index i ON ((c.oid = i.indrelid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE (c.relkind = ANY (ARRAY['r'::"char", 't'::"char"])) GROUP BY c.oid, n.nspname, c.relname; pg_stat_bgwriter | SELECT pg_stat_get_bgwriter_timed_checkpoints() AS checkpoints_timed, pg_stat_get_bgwriter_requested_checkpoints() AS checkpoints_req, pg_stat_get_bgwriter_buf_written_checkpoints() AS buffers_checkpoint, pg_stat_get_bgwriter_buf_written_clean() AS buffers_clean, pg_stat_get_bgwriter_maxwritten_clean() AS maxwritten_clean, pg_stat_get_buf_written_backend() AS buffers_backend, pg_stat_get_buf_fsync_backend() AS buffers_backend_fsync, pg_stat_get_buf_alloc() AS buffers_alloc, pg_stat_get_bgwriter_stat_reset_time() AS stats_reset; - pg_stat_database | SELECT d.oid AS datid, d.datname, pg_stat_get_db_numbackends(d.oid) AS numbackends, pg_stat_get_db_xact_commit(d.oid) AS xact_commit, pg_stat_get_db_xact_rollback(d.oid) AS xact_rollback, (pg_stat_get_db_blocks_fetched(d.oid) - pg_stat_get_db_blocks_hit(d.oid)) AS blks_read, pg_stat_get_db_blocks_hit(d.oid) AS blks_hit, pg_stat_get_db_tuples_returned(d.oid) AS tup_returned, pg_stat_get_db_tuples_fetched(d.oid) AS tup_fetched, pg_stat_get_db_tuples_inserted(d.oid) AS tup_inserted, pg_stat_get_db_tuples_updated(d.oid) AS tup_updated, pg_stat_get_db_tuples_deleted(d.oid) AS tup_deleted, pg_stat_get_db_conflict_all(d.oid) AS conflicts, pg_stat_get_db_temp_files(d.oid) AS temp_files, pg_stat_get_db_temp_bytes(d.oid) AS temp_bytes, pg_stat_get_db_deadlocks(d.oid) AS deadlocks, pg_stat_get_db_stat_reset_time(d.oid) AS stats_reset FROM pg_database d; + pg_stat_database | SELECT d.oid AS datid, d.datname, pg_stat_get_db_numbackends(d.oid) AS numbackends, pg_stat_get_db_xact_commit(d.oid) AS xact_commit, pg_stat_get_db_xact_rollback(d.oid) AS xact_rollback, (pg_stat_get_db_blocks_fetched(d.oid) - pg_stat_get_db_blocks_hit(d.oid)) AS blks_read, pg_stat_get_db_blocks_hit(d.oid) AS blks_hit, pg_stat_get_db_blocks_time(d.oid) AS blks_time, pg_stat_get_db_tuples_returned(d.oid) AS tup_returned, pg_stat_get_db_tuples_fetched(d.oid) AS tup_fetched, pg_stat_get_db_tuples_inserted(d.oid) AS tup_inserted, pg_stat_get_db_tuples_updated(d.oid) AS tup_updated, pg_stat_get_db_tuples_deleted(d.oid) AS tup_deleted, pg_stat_get_db_conflict_all(d.oid) AS conflicts, pg_stat_get_db_temp_files(d.oid) AS temp_files, pg_stat_get_db_temp_bytes(d.oid) AS temp_bytes, pg_stat_get_db_deadlocks(d.oid) AS deadlocks, pg_stat_get_db_stat_reset_time(d.oid) AS stats_reset FROM pg_database d; pg_stat_database_conflicts | SELECT d.oid AS datid, d.datname, pg_stat_get_db_conflict_tablespace(d.oid) AS confl_tablespace, pg_stat_get_db_conflict_lock(d.oid) AS confl_lock, pg_stat_get_db_conflict_snapshot(d.oid) AS confl_snapshot, pg_stat_get_db_conflict_bufferpin(d.oid) AS confl_bufferpin, pg_stat_get_db_conflict_startup_deadlock(d.oid) AS confl_deadlock FROM pg_database d; pg_stat_replication | SELECT s.pid, s.usesysid, u.rolname AS usename, s.application_name, s.client_addr, s.client_hostname, s.client_port, s.backend_start, w.state, w.sent_location, w.write_location, w.flush_location, w.replay_location, w.sync_priority, w.sync_state FROM pg_stat_get_activity(NULL::integer) s(datid, pid, usesysid, application_name, state, query, waiting, xact_start, query_start, backend_start, state_change, client_addr, client_hostname, client_port), pg_authid u, pg_stat_get_wal_senders() w(pid, state, sent_location, write_location, flush_location, replay_location, sync_priority, sync_state) WHERE ((s.usesysid = u.oid) AND (s.pid = w.pid)); pg_stat_sys_indexes | SELECT pg_stat_all_indexes.relid, pg_stat_all_indexes.indexrelid, pg_stat_all_indexes.schemaname, pg_stat_all_indexes.relname, pg_stat_all_indexes.indexrelname, pg_stat_all_indexes.idx_scan, pg_stat_all_indexes.idx_tup_read, pg_stat_all_indexes.idx_tup_fetch FROM pg_stat_all_indexes WHERE ((pg_stat_all_indexes.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_stat_all_indexes.schemaname ~ '^pg_toast'::text)); @@ -1308,15 +1308,15 @@ SELECT viewname, definition FROM pg_views WHERE schemaname <> 'information_schem pg_stat_xact_sys_tables | SELECT pg_stat_xact_all_tables.relid, pg_stat_xact_all_tables.schemaname, pg_stat_xact_all_tables.relname, pg_stat_xact_all_tables.seq_scan, pg_stat_xact_all_tables.seq_tup_read, pg_stat_xact_all_tables.idx_scan, pg_stat_xact_all_tables.idx_tup_fetch, pg_stat_xact_all_tables.n_tup_ins, pg_stat_xact_all_tables.n_tup_upd, pg_stat_xact_all_tables.n_tup_del, pg_stat_xact_all_tables.n_tup_hot_upd FROM pg_stat_xact_all_tables WHERE ((pg_stat_xact_all_tables.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_stat_xact_all_tables.schemaname ~ '^pg_toast'::text)); pg_stat_xact_user_functions | SELECT p.oid AS funcid, n.nspname AS schemaname, p.proname AS funcname, pg_stat_get_xact_function_calls(p.oid) AS calls, (pg_stat_get_xact_function_time(p.oid) / 1000) AS total_time, (pg_stat_get_xact_function_self_time(p.oid) / 1000) AS self_time FROM (pg_proc p LEFT JOIN pg_namespace n ON ((n.oid = p.pronamespace))) WHERE ((p.prolang <> (12)::oid) AND (pg_stat_get_xact_function_calls(p.oid) IS NOT NULL)); pg_stat_xact_user_tables | SELECT pg_stat_xact_all_tables.relid, pg_stat_xact_all_tables.schemaname, pg_stat_xact_all_tables.relname, pg_stat_xact_all_tables.seq_scan, pg_stat_xact_all_tables.seq_tup_read, pg_stat_xact_all_tables.idx_scan, pg_stat_xact_all_tables.idx_tup_fetch, pg_stat_xact_all_tables.n_tup_ins, pg_stat_xact_all_tables.n_tup_upd, pg_stat_xact_all_tables.n_tup_del, pg_stat_xact_all_tables.n_tup_hot_upd FROM pg_stat_xact_all_tables WHERE ((pg_stat_xact_all_tables.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_stat_xact_all_tables.schemaname !~ '^pg_toast'::text)); - pg_statio_all_indexes | SELECT c.oid AS relid, i.oid AS indexrelid, n.nspname AS schemaname, c.relname, i.relname AS indexrelname, (pg_stat_get_blocks_fetched(i.oid) - pg_stat_get_blocks_hit(i.oid)) AS idx_blks_read, pg_stat_get_blocks_hit(i.oid) AS idx_blks_hit FROM (((pg_class c JOIN pg_index x ON ((c.oid = x.indrelid))) JOIN pg_class i ON ((i.oid = x.indexrelid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE (c.relkind = ANY (ARRAY['r'::"char", 't'::"char"])); - pg_statio_all_sequences | SELECT c.oid AS relid, n.nspname AS schemaname, c.relname, (pg_stat_get_blocks_fetched(c.oid) - pg_stat_get_blocks_hit(c.oid)) AS blks_read, pg_stat_get_blocks_hit(c.oid) AS blks_hit FROM (pg_class c LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE (c.relkind = 'S'::"char"); - pg_statio_all_tables | SELECT c.oid AS relid, n.nspname AS schemaname, c.relname, (pg_stat_get_blocks_fetched(c.oid) - pg_stat_get_blocks_hit(c.oid)) AS heap_blks_read, pg_stat_get_blocks_hit(c.oid) AS heap_blks_hit, (sum((pg_stat_get_blocks_fetched(i.indexrelid) - pg_stat_get_blocks_hit(i.indexrelid))))::bigint AS idx_blks_read, (sum(pg_stat_get_blocks_hit(i.indexrelid)))::bigint AS idx_blks_hit, (pg_stat_get_blocks_fetched(t.oid) - pg_stat_get_blocks_hit(t.oid)) AS toast_blks_read, pg_stat_get_blocks_hit(t.oid) AS toast_blks_hit, (pg_stat_get_blocks_fetched(x.oid) - pg_stat_get_blocks_hit(x.oid)) AS tidx_blks_read, pg_stat_get_blocks_hit(x.oid) AS tidx_blks_hit FROM ((((pg_class c LEFT JOIN pg_index i ON ((c.oid = i.indrelid))) LEFT JOIN pg_class t ON ((c.reltoastrelid = t.oid))) LEFT JOIN pg_class x ON ((t.reltoastidxid = x.oid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE (c.relkind = ANY (ARRAY['r'::"char", 't'::"char"])) GROUP BY c.oid, n.nspname, c.relname, t.oid, x.oid; - pg_statio_sys_indexes | SELECT pg_statio_all_indexes.relid, pg_statio_all_indexes.indexrelid, pg_statio_all_indexes.schemaname, pg_statio_all_indexes.relname, pg_statio_all_indexes.indexrelname, pg_statio_all_indexes.idx_blks_read, pg_statio_all_indexes.idx_blks_hit FROM pg_statio_all_indexes WHERE ((pg_statio_all_indexes.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_statio_all_indexes.schemaname ~ '^pg_toast'::text)); - pg_statio_sys_sequences | SELECT pg_statio_all_sequences.relid, pg_statio_all_sequences.schemaname, pg_statio_all_sequences.relname, pg_statio_all_sequences.blks_read, pg_statio_all_sequences.blks_hit FROM pg_statio_all_sequences WHERE ((pg_statio_all_sequences.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_statio_all_sequences.schemaname ~ '^pg_toast'::text)); - pg_statio_sys_tables | SELECT pg_statio_all_tables.relid, pg_statio_all_tables.schemaname, pg_statio_all_tables.relname, pg_statio_all_tables.heap_blks_read, pg_statio_all_tables.heap_blks_hit, pg_statio_all_tables.idx_blks_read, pg_statio_all_tables.idx_blks_hit, pg_statio_all_tables.toast_blks_read, pg_statio_all_tables.toast_blks_hit, pg_statio_all_tables.tidx_blks_read, pg_statio_all_tables.tidx_blks_hit FROM pg_statio_all_tables WHERE ((pg_statio_all_tables.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_statio_all_tables.schemaname ~ '^pg_toast'::text)); - pg_statio_user_indexes | SELECT pg_statio_all_indexes.relid, pg_statio_all_indexes.indexrelid, pg_statio_all_indexes.schemaname, pg_statio_all_indexes.relname, pg_statio_all_indexes.indexrelname, pg_statio_all_indexes.idx_blks_read, pg_statio_all_indexes.idx_blks_hit FROM pg_statio_all_indexes WHERE ((pg_statio_all_indexes.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_statio_all_indexes.schemaname !~ '^pg_toast'::text)); - pg_statio_user_sequences | SELECT pg_statio_all_sequences.relid, pg_statio_all_sequences.schemaname, pg_statio_all_sequences.relname, pg_statio_all_sequences.blks_read, pg_statio_all_sequences.blks_hit FROM pg_statio_all_sequences WHERE ((pg_statio_all_sequences.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_statio_all_sequences.schemaname !~ '^pg_toast'::text)); - pg_statio_user_tables | SELECT pg_statio_all_tables.relid, pg_statio_all_tables.schemaname, pg_statio_all_tables.relname, pg_statio_all_tables.heap_blks_read, pg_statio_all_tables.heap_blks_hit, pg_statio_all_tables.idx_blks_read, pg_statio_all_tables.idx_blks_hit, pg_statio_all_tables.toast_blks_read, pg_statio_all_tables.toast_blks_hit, pg_statio_all_tables.tidx_blks_read, pg_statio_all_tables.tidx_blks_hit FROM pg_statio_all_tables WHERE ((pg_statio_all_tables.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_statio_all_tables.schemaname !~ '^pg_toast'::text)); + pg_statio_all_indexes | SELECT c.oid AS relid, i.oid AS indexrelid, n.nspname AS schemaname, c.relname, i.relname AS indexrelname, (pg_stat_get_blocks_fetched(i.oid) - pg_stat_get_blocks_hit(i.oid)) AS idx_blks_read, pg_stat_get_blocks_hit(i.oid) AS idx_blks_hit, pg_stat_get_blocks_time(i.oid) AS idx_blks_time FROM (((pg_class c JOIN pg_index x ON ((c.oid = x.indrelid))) JOIN pg_class i ON ((i.oid = x.indexrelid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE (c.relkind = ANY (ARRAY['r'::"char", 't'::"char"])); + pg_statio_all_sequences | SELECT c.oid AS relid, n.nspname AS schemaname, c.relname, (pg_stat_get_blocks_fetched(c.oid) - pg_stat_get_blocks_hit(c.oid)) AS blks_read, pg_stat_get_blocks_hit(c.oid) AS blks_hit, pg_stat_get_blocks_time(c.oid) AS blks_time FROM (pg_class c LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE (c.relkind = 'S'::"char"); + pg_statio_all_tables | SELECT c.oid AS relid, n.nspname AS schemaname, c.relname, (pg_stat_get_blocks_fetched(c.oid) - pg_stat_get_blocks_hit(c.oid)) AS heap_blks_read, pg_stat_get_blocks_hit(c.oid) AS heap_blks_hit, pg_stat_get_blocks_time(c.oid) AS heap_blks_time, (sum((pg_stat_get_blocks_fetched(i.indexrelid) - pg_stat_get_blocks_hit(i.indexrelid))))::bigint AS idx_blks_read, (sum(pg_stat_get_blocks_hit(i.indexrelid)))::bigint AS idx_blks_hit, (sum(pg_stat_get_blocks_time(i.indexrelid)))::bigint AS idx_blks_time, (pg_stat_get_blocks_fetched(t.oid) - pg_stat_get_blocks_hit(t.oid)) AS toast_blks_read, pg_stat_get_blocks_hit(t.oid) AS toast_blks_hit, pg_stat_get_blocks_time(t.oid) AS toast_blks_time, (pg_stat_get_blocks_fetched(x.oid) - pg_stat_get_blocks_hit(x.oid)) AS tidx_blks_read, pg_stat_get_blocks_hit(x.oid) AS tidx_blks_hit, pg_stat_get_blocks_time(x.oid) AS tidx_blks_time FROM ((((pg_class c LEFT JOIN pg_index i ON ((c.oid = i.indrelid))) LEFT JOIN pg_class t ON ((c.reltoastrelid = t.oid))) LEFT JOIN pg_class x ON ((t.reltoastidxid = x.oid))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE (c.relkind = ANY (ARRAY['r'::"char", 't'::"char"])) GROUP BY c.oid, n.nspname, c.relname, t.oid, x.oid; + pg_statio_sys_indexes | SELECT pg_statio_all_indexes.relid, pg_statio_all_indexes.indexrelid, pg_statio_all_indexes.schemaname, pg_statio_all_indexes.relname, pg_statio_all_indexes.indexrelname, pg_statio_all_indexes.idx_blks_read, pg_statio_all_indexes.idx_blks_hit, pg_statio_all_indexes.idx_blks_time FROM pg_statio_all_indexes WHERE ((pg_statio_all_indexes.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_statio_all_indexes.schemaname ~ '^pg_toast'::text)); + pg_statio_sys_sequences | SELECT pg_statio_all_sequences.relid, pg_statio_all_sequences.schemaname, pg_statio_all_sequences.relname, pg_statio_all_sequences.blks_read, pg_statio_all_sequences.blks_hit, pg_statio_all_sequences.blks_time FROM pg_statio_all_sequences WHERE ((pg_statio_all_sequences.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_statio_all_sequences.schemaname ~ '^pg_toast'::text)); + pg_statio_sys_tables | SELECT pg_statio_all_tables.relid, pg_statio_all_tables.schemaname, pg_statio_all_tables.relname, pg_statio_all_tables.heap_blks_read, pg_statio_all_tables.heap_blks_hit, pg_statio_all_tables.heap_blks_time, pg_statio_all_tables.idx_blks_read, pg_statio_all_tables.idx_blks_hit, pg_statio_all_tables.idx_blks_time, pg_statio_all_tables.toast_blks_read, pg_statio_all_tables.toast_blks_hit, pg_statio_all_tables.toast_blks_time, pg_statio_all_tables.tidx_blks_read, pg_statio_all_tables.tidx_blks_hit, pg_statio_all_tables.tidx_blks_time FROM pg_statio_all_tables WHERE ((pg_statio_all_tables.schemaname = ANY (ARRAY['pg_catalog'::name, 'information_schema'::name])) OR (pg_statio_all_tables.schemaname ~ '^pg_toast'::text)); + pg_statio_user_indexes | SELECT pg_statio_all_indexes.relid, pg_statio_all_indexes.indexrelid, pg_statio_all_indexes.schemaname, pg_statio_all_indexes.relname, pg_statio_all_indexes.indexrelname, pg_statio_all_indexes.idx_blks_read, pg_statio_all_indexes.idx_blks_hit, pg_statio_all_indexes.idx_blks_time FROM pg_statio_all_indexes WHERE ((pg_statio_all_indexes.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_statio_all_indexes.schemaname !~ '^pg_toast'::text)); + pg_statio_user_sequences | SELECT pg_statio_all_sequences.relid, pg_statio_all_sequences.schemaname, pg_statio_all_sequences.relname, pg_statio_all_sequences.blks_read, pg_statio_all_sequences.blks_hit, pg_statio_all_sequences.blks_time FROM pg_statio_all_sequences WHERE ((pg_statio_all_sequences.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_statio_all_sequences.schemaname !~ '^pg_toast'::text)); + pg_statio_user_tables | SELECT pg_statio_all_tables.relid, pg_statio_all_tables.schemaname, pg_statio_all_tables.relname, pg_statio_all_tables.heap_blks_read, pg_statio_all_tables.heap_blks_hit, pg_statio_all_tables.heap_blks_time, pg_statio_all_tables.idx_blks_read, pg_statio_all_tables.idx_blks_hit, pg_statio_all_tables.idx_blks_time, pg_statio_all_tables.toast_blks_read, pg_statio_all_tables.toast_blks_hit, pg_statio_all_tables.toast_blks_time, pg_statio_all_tables.tidx_blks_read, pg_statio_all_tables.tidx_blks_hit, pg_statio_all_tables.tidx_blks_time FROM pg_statio_all_tables WHERE ((pg_statio_all_tables.schemaname <> ALL (ARRAY['pg_catalog'::name, 'information_schema'::name])) AND (pg_statio_all_tables.schemaname !~ '^pg_toast'::text)); pg_stats | SELECT n.nspname AS schemaname, c.relname AS tablename, a.attname, s.stainherit AS inherited, s.stanullfrac AS null_frac, s.stawidth AS avg_width, s.stadistinct AS n_distinct, CASE WHEN (s.stakind1 = 1) THEN s.stavalues1 WHEN (s.stakind2 = 1) THEN s.stavalues2 WHEN (s.stakind3 = 1) THEN s.stavalues3 WHEN (s.stakind4 = 1) THEN s.stavalues4 WHEN (s.stakind5 = 1) THEN s.stavalues5 ELSE NULL::anyarray END AS most_common_vals, CASE WHEN (s.stakind1 = 1) THEN s.stanumbers1 WHEN (s.stakind2 = 1) THEN s.stanumbers2 WHEN (s.stakind3 = 1) THEN s.stanumbers3 WHEN (s.stakind4 = 1) THEN s.stanumbers4 WHEN (s.stakind5 = 1) THEN s.stanumbers5 ELSE NULL::real[] END AS most_common_freqs, CASE WHEN (s.stakind1 = 2) THEN s.stavalues1 WHEN (s.stakind2 = 2) THEN s.stavalues2 WHEN (s.stakind3 = 2) THEN s.stavalues3 WHEN (s.stakind4 = 2) THEN s.stavalues4 WHEN (s.stakind5 = 2) THEN s.stavalues5 ELSE NULL::anyarray END AS histogram_bounds, CASE WHEN (s.stakind1 = 3) THEN s.stanumbers1[1] WHEN (s.stakind2 = 3) THEN s.stanumbers2[1] WHEN (s.stakind3 = 3) THEN s.stanumbers3[1] WHEN (s.stakind4 = 3) THEN s.stanumbers4[1] WHEN (s.stakind5 = 3) THEN s.stanumbers5[1] ELSE NULL::real END AS correlation, CASE WHEN (s.stakind1 = 4) THEN s.stavalues1 WHEN (s.stakind2 = 4) THEN s.stavalues2 WHEN (s.stakind3 = 4) THEN s.stavalues3 WHEN (s.stakind4 = 4) THEN s.stavalues4 WHEN (s.stakind5 = 4) THEN s.stavalues5 ELSE NULL::anyarray END AS most_common_elems, CASE WHEN (s.stakind1 = 4) THEN s.stanumbers1 WHEN (s.stakind2 = 4) THEN s.stanumbers2 WHEN (s.stakind3 = 4) THEN s.stanumbers3 WHEN (s.stakind4 = 4) THEN s.stanumbers4 WHEN (s.stakind5 = 4) THEN s.stanumbers5 ELSE NULL::real[] END AS most_common_elem_freqs, CASE WHEN (s.stakind1 = 5) THEN s.stanumbers1 WHEN (s.stakind2 = 5) THEN s.stanumbers2 WHEN (s.stakind3 = 5) THEN s.stanumbers3 WHEN (s.stakind4 = 5) THEN s.stanumbers4 WHEN (s.stakind5 = 5) THEN s.stanumbers5 ELSE NULL::real[] END AS elem_count_histogram FROM (((pg_statistic s JOIN pg_class c ON ((c.oid = s.starelid))) JOIN pg_attribute a ON (((c.oid = a.attrelid) AND (a.attnum = s.staattnum)))) LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) WHERE ((NOT a.attisdropped) AND has_column_privilege(c.oid, a.attnum, 'select'::text)); pg_tables | SELECT n.nspname AS schemaname, c.relname AS tablename, pg_get_userbyid(c.relowner) AS tableowner, t.spcname AS tablespace, c.relhasindex AS hasindexes, c.relhasrules AS hasrules, c.relhastriggers AS hastriggers FROM ((pg_class c LEFT JOIN pg_namespace n ON ((n.oid = c.relnamespace))) LEFT JOIN pg_tablespace t ON ((t.oid = c.reltablespace))) WHERE (c.relkind = 'r'::"char"); pg_timezone_abbrevs | SELECT pg_timezone_abbrevs.abbrev, pg_timezone_abbrevs.utc_offset, pg_timezone_abbrevs.is_dst FROM pg_timezone_abbrevs() pg_timezone_abbrevs(abbrev, utc_offset, is_dst); [text/x-patch] io-stats-statement.v5.patch (7.6K, ../../CA+CSw_tfw+Vcgy1WXVx5kt5=739D0_aTXxzh5Ubf-SABDASYqg@mail.gmail.com/3-io-stats-statement.v5.patch) download | inline diff: diff --git a/contrib/pg_stat_statements/pg_stat_statements--1.0--1.1.sql b/contrib/pg_stat_statements/pg_stat_statements--1.0--1.1.sql index 223271d..f976419 100644 --- a/contrib/pg_stat_statements/pg_stat_statements--1.0--1.1.sql +++ b/contrib/pg_stat_statements/pg_stat_statements--1.0--1.1.sql @@ -28,7 +28,9 @@ CREATE FUNCTION pg_stat_statements( OUT local_blks_dirtied int8, OUT local_blks_written int8, OUT temp_blks_read int8, - OUT temp_blks_written int8 + OUT temp_blks_written int8, + OUT time_read float8, + OUT time_write float8 ) RETURNS SETOF record AS 'MODULE_PATHNAME' diff --git a/contrib/pg_stat_statements/pg_stat_statements--1.1.sql b/contrib/pg_stat_statements/pg_stat_statements--1.1.sql index 1233736..f4bdf12 100644 --- a/contrib/pg_stat_statements/pg_stat_statements--1.1.sql +++ b/contrib/pg_stat_statements/pg_stat_statements--1.1.sql @@ -25,7 +25,9 @@ CREATE FUNCTION pg_stat_statements( OUT local_blks_dirtied int8, OUT local_blks_written int8, OUT temp_blks_read int8, - OUT temp_blks_written int8 + OUT temp_blks_written int8, + OUT time_read float8, + OUT time_write float8 ) RETURNS SETOF record AS 'MODULE_PATHNAME' diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c index 914fbf2..8e34740 100644 --- a/contrib/pg_stat_statements/pg_stat_statements.c +++ b/contrib/pg_stat_statements/pg_stat_statements.c @@ -85,6 +85,8 @@ typedef struct Counters int64 local_blks_written; /* # of local disk blocks written */ int64 temp_blks_read; /* # of temp blocks read */ int64 temp_blks_written; /* # of temp blocks written */ + double time_read; /* time spent reading in seconds */ + double time_write; /* time spent writing in seconds */ double usage; /* usage factor */ } Counters; @@ -618,9 +620,9 @@ pgss_ProcessUtility(Node *parsetree, const char *queryString, instr_time start; instr_time duration; uint64 rows = 0; - BufferUsage bufusage; + BufferUsage bufusage_start, bufusage; - bufusage = pgBufferUsage; + bufusage_start = pgBufferUsage; INSTR_TIME_SET_CURRENT(start); nested_level++; @@ -651,25 +653,29 @@ pgss_ProcessUtility(Node *parsetree, const char *queryString, /* calc differences of buffer counters. */ bufusage.shared_blks_hit = - pgBufferUsage.shared_blks_hit - bufusage.shared_blks_hit; + pgBufferUsage.shared_blks_hit - bufusage_start.shared_blks_hit; bufusage.shared_blks_read = - pgBufferUsage.shared_blks_read - bufusage.shared_blks_read; + pgBufferUsage.shared_blks_read - bufusage_start.shared_blks_read; bufusage.shared_blks_dirtied = - pgBufferUsage.shared_blks_dirtied - bufusage.shared_blks_dirtied; + pgBufferUsage.shared_blks_dirtied - bufusage_start.shared_blks_dirtied; bufusage.shared_blks_written = - pgBufferUsage.shared_blks_written - bufusage.shared_blks_written; + pgBufferUsage.shared_blks_written - bufusage_start.shared_blks_written; bufusage.local_blks_hit = - pgBufferUsage.local_blks_hit - bufusage.local_blks_hit; + pgBufferUsage.local_blks_hit - bufusage_start.local_blks_hit; bufusage.local_blks_read = - pgBufferUsage.local_blks_read - bufusage.local_blks_read; + pgBufferUsage.local_blks_read - bufusage_start.local_blks_read; bufusage.local_blks_dirtied = - pgBufferUsage.local_blks_dirtied - bufusage.local_blks_dirtied; + pgBufferUsage.local_blks_dirtied - bufusage_start.local_blks_dirtied; bufusage.local_blks_written = - pgBufferUsage.local_blks_written - bufusage.local_blks_written; + pgBufferUsage.local_blks_written - bufusage_start.local_blks_written; bufusage.temp_blks_read = - pgBufferUsage.temp_blks_read - bufusage.temp_blks_read; + pgBufferUsage.temp_blks_read - bufusage_start.temp_blks_read; bufusage.temp_blks_written = - pgBufferUsage.temp_blks_written - bufusage.temp_blks_written; + pgBufferUsage.temp_blks_written - bufusage_start.temp_blks_written; + bufusage.time_read = pgBufferUsage.time_read; + INSTR_TIME_SUBTRACT(bufusage.time_read, bufusage_start.time_read); + bufusage.time_write = pgBufferUsage.time_write; + INSTR_TIME_SUBTRACT(bufusage.time_write, bufusage_start.time_write); pgss_store(queryString, INSTR_TIME_GET_DOUBLE(duration), rows, &bufusage); @@ -780,6 +786,8 @@ pgss_store(const char *query, double total_time, uint64 rows, e->counters.local_blks_written += bufusage->local_blks_written; e->counters.temp_blks_read += bufusage->temp_blks_read; e->counters.temp_blks_written += bufusage->temp_blks_written; + e->counters.time_read += INSTR_TIME_GET_DOUBLE(bufusage->time_read); + e->counters.time_write += INSTR_TIME_GET_DOUBLE(bufusage->time_write); e->counters.usage += usage; SpinLockRelease(&e->mutex); } @@ -802,7 +810,7 @@ pg_stat_statements_reset(PG_FUNCTION_ARGS) } #define PG_STAT_STATEMENTS_COLS_V1_0 14 -#define PG_STAT_STATEMENTS_COLS 16 +#define PG_STAT_STATEMENTS_COLS 18 /* * Retrieve statement statistics. @@ -819,7 +827,7 @@ pg_stat_statements(PG_FUNCTION_ARGS) bool is_superuser = superuser(); HASH_SEQ_STATUS hash_seq; pgssEntry *entry; - bool sql_supports_dirty_counters = true; + bool sql_supports_v1_1_counters = true; if (!pgss || !pgss_hash) ereport(ERROR, @@ -841,7 +849,7 @@ pg_stat_statements(PG_FUNCTION_ARGS) if (get_call_result_type(fcinfo, NULL, &tupdesc) != TYPEFUNC_COMPOSITE) elog(ERROR, "return type must be a row type"); if (tupdesc->natts == PG_STAT_STATEMENTS_COLS_V1_0) - sql_supports_dirty_counters = false; + sql_supports_v1_1_counters = false; per_query_ctx = rsinfo->econtext->ecxt_per_query_memory; oldcontext = MemoryContextSwitchTo(per_query_ctx); @@ -899,18 +907,22 @@ pg_stat_statements(PG_FUNCTION_ARGS) values[i++] = Int64GetDatumFast(tmp.rows); values[i++] = Int64GetDatumFast(tmp.shared_blks_hit); values[i++] = Int64GetDatumFast(tmp.shared_blks_read); - if (sql_supports_dirty_counters) + if (sql_supports_v1_1_counters) values[i++] = Int64GetDatumFast(tmp.shared_blks_dirtied); values[i++] = Int64GetDatumFast(tmp.shared_blks_written); values[i++] = Int64GetDatumFast(tmp.local_blks_hit); values[i++] = Int64GetDatumFast(tmp.local_blks_read); - if (sql_supports_dirty_counters) + if (sql_supports_v1_1_counters) values[i++] = Int64GetDatumFast(tmp.local_blks_dirtied); values[i++] = Int64GetDatumFast(tmp.local_blks_written); values[i++] = Int64GetDatumFast(tmp.temp_blks_read); values[i++] = Int64GetDatumFast(tmp.temp_blks_written); + if (sql_supports_v1_1_counters) { + values[i++] = Float8GetDatumFast(tmp.time_read); + values[i++] = Float8GetDatumFast(tmp.time_write); + } - Assert(i == sql_supports_dirty_counters ? \ + Assert(i == sql_supports_v1_1_counters ? \ PG_STAT_STATEMENTS_COLS : PG_STAT_STATEMENTS_COLS_V1_0); tuplestore_putvalues(tupstore, tupdesc, values, nulls); diff --git a/doc/src/sgml/pgstatstatements.sgml b/doc/src/sgml/pgstatstatements.sgml index ab34ca1..f9da247 100644 --- a/doc/src/sgml/pgstatstatements.sgml +++ b/doc/src/sgml/pgstatstatements.sgml @@ -155,6 +155,20 @@ <entry>Total number of temp blocks writes by the statement</entry> </row> + <row> + <entry><structfield>time_read</structfield></entry> + <entry><type>double precision</type></entry> + <entry></entry> + <entry>Total time spent by the statement for reading blocks, in seconds</entry> + </row> + + <row> + <entry><structfield>time_write</structfield></entry> + <entry><type>double precision</type></entry> + <entry></entry> + <entry>Total time spent by the statement for writing out dirty blocks, in seconds</entry> + </row> + </tbody> </tgroup> </table> ^ permalink raw reply [nested|flat] 12552+ messages in thread
* Re: Patch: add timing of buffer I/O requests 2012-03-21 15:01 Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-22 23:38 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> @ 2012-03-27 18:58 ` Robert Haas <[email protected]> 2012-03-27 19:22 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> 2012-03-27 19:23 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-28 00:41 ` Re: Patch: add timing of buffer I/O requests Greg Stark <[email protected]> 0 siblings, 3 replies; 12552+ messages in thread From: Robert Haas @ 2012-03-27 18:58 UTC (permalink / raw) To: Ants Aasma <[email protected]>; +Cc: Greg Smith <[email protected]>; pgsql-hackers On Thu, Mar 22, 2012 at 7:38 PM, Ants Aasma <[email protected]> wrote: > On Wed, Mar 21, 2012 at 5:01 PM, Robert Haas <[email protected]> wrote: >> This seems to have bitrotted again. :-( >> >> Can you please rebase again? > > Rebased. I've committed the core of this. I left out the stats collector stuff, because it's still per-table and I think perhaps we should back off to just per-database. I changed it so that it does not conflate wait time with I/O time. Maybe there should be a separate method of measuring wait time, but I don't think it's a good idea for the per-backend stats to measure a different thing than what gets reported up to the stats collector - we should have ONE definition of each counter. I also tweaked the EXPLAIN output format a bit, and the docs. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 12552+ messages in thread
* Re: Patch: add timing of buffer I/O requests 2012-03-21 15:01 Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-22 23:38 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> 2012-03-27 18:58 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> @ 2012-03-27 19:22 ` Ants Aasma <[email protected]> 2012-03-28 00:30 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2 siblings, 1 reply; 12552+ messages in thread From: Ants Aasma @ 2012-03-27 19:22 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Greg Smith <[email protected]>; pgsql-hackers On Tue, Mar 27, 2012 at 9:58 PM, Robert Haas <[email protected]> wrote: > I've committed the core of this. I left out the stats collector > stuff, because it's still per-table and I think perhaps we should back > off to just per-database. I changed it so that it does not conflate > wait time with I/O time. Maybe there should be a separate method of > measuring wait time, but I don't think it's a good idea for the > per-backend stats to measure a different thing than what gets reported > up to the stats collector - we should have ONE definition of each > counter. I also tweaked the EXPLAIN output format a bit, and the > docs. Thank you for working on this. Taking a fresh look at it, I agree with you that conflating waiting for backend local IO, and IO performed by some other backend might paint us into a corner. For most workloads the wait for IO performed by others should be the minority anyway. I won't really miss the per table stats. But if the pg_stat_statements normalisation patch gets commited, it would be really neat to also have IO waits there. It would be much easier to quickly diagnose "what is causing all this IO" issues. Thanks again, Ants Aasma -- Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de ^ permalink raw reply [nested|flat] 12552+ messages in thread
* Re: Patch: add timing of buffer I/O requests 2012-03-21 15:01 Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-22 23:38 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> 2012-03-27 18:58 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-27 19:22 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> @ 2012-03-28 00:30 ` Robert Haas <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Robert Haas @ 2012-03-28 00:30 UTC (permalink / raw) To: Ants Aasma <[email protected]>; +Cc: Greg Smith <[email protected]>; pgsql-hackers On Tue, Mar 27, 2012 at 3:22 PM, Ants Aasma <[email protected]> wrote: > On Tue, Mar 27, 2012 at 9:58 PM, Robert Haas <[email protected]> wrote: >> I've committed the core of this. I left out the stats collector >> stuff, because it's still per-table and I think perhaps we should back >> off to just per-database. I changed it so that it does not conflate >> wait time with I/O time. Maybe there should be a separate method of >> measuring wait time, but I don't think it's a good idea for the >> per-backend stats to measure a different thing than what gets reported >> up to the stats collector - we should have ONE definition of each >> counter. I also tweaked the EXPLAIN output format a bit, and the >> docs. > > Thank you for working on this. > > Taking a fresh look at it, I agree with you that conflating waiting > for backend local IO, and IO performed by some other backend might > paint us into a corner. For most workloads the wait for IO performed > by others should be the minority anyway. > > I won't really miss the per table stats. But if the pg_stat_statements > normalisation patch gets commited, it would be really neat to also > have IO waits there. It would be much easier to quickly diagnose "what > is causing all this IO" issues. So, the pg_stat_statements part of this is committed now. Do you want to prepare a revised patch to add per-database counters to the statistics collector? I think that might be a good idea... -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 12552+ messages in thread
* Re: Patch: add timing of buffer I/O requests 2012-03-21 15:01 Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-22 23:38 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> 2012-03-27 18:58 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> @ 2012-03-27 19:23 ` Robert Haas <[email protected]> 2 siblings, 0 replies; 12552+ messages in thread From: Robert Haas @ 2012-03-27 19:23 UTC (permalink / raw) To: Ants Aasma <[email protected]>; +Cc: Greg Smith <[email protected]>; pgsql-hackers On Tue, Mar 27, 2012 at 2:58 PM, Robert Haas <[email protected]> wrote: > On Thu, Mar 22, 2012 at 7:38 PM, Ants Aasma <[email protected]> wrote: >> On Wed, Mar 21, 2012 at 5:01 PM, Robert Haas <[email protected]> wrote: >>> This seems to have bitrotted again. :-( >>> >>> Can you please rebase again? >> >> Rebased. > > I've committed the core of this. I left out the stats collector > stuff, because it's still per-table and I think perhaps we should back > off to just per-database. I changed it so that it does not conflate > wait time with I/O time. Maybe there should be a separate method of > measuring wait time, but I don't think it's a good idea for the > per-backend stats to measure a different thing than what gets reported > up to the stats collector - we should have ONE definition of each > counter. I also tweaked the EXPLAIN output format a bit, and the > docs. And I've now committed the pg_stat_statements code as well, hopefully not stomping too badly one what Tom's apparently in the midst of doing with Peter's pg_stat_statements patch. I committed this almost exactly as submitted; just a minor code style correction and a few documentation enhancements. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 12552+ messages in thread
* Re: Patch: add timing of buffer I/O requests 2012-03-21 15:01 Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-22 23:38 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> 2012-03-27 18:58 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> @ 2012-03-28 00:41 ` Greg Stark <[email protected]> 2012-03-28 00:51 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2 siblings, 1 reply; 12552+ messages in thread From: Greg Stark @ 2012-03-28 00:41 UTC (permalink / raw) To: Robert Haas <[email protected]>; +Cc: Ants Aasma <[email protected]>; Greg Smith <[email protected]>; pgsql-hackers On Tue, Mar 27, 2012 at 7:58 PM, Robert Haas <[email protected]> wrote: > I've committed the core of this. I left out the stats collector > stuff, because it's still per-table and I think perhaps we should back > off to just per-database. I changed it so that it does not conflate > wait time with I/O time. Maybe there should be a separate method of > measuring wait time, but I don't think it's a good idea for the > per-backend stats to measure a different thing than what gets reported > up to the stats collector - we should have ONE definition of each > counter. I also tweaked the EXPLAIN output format a bit, and the > docs. Maybe I missed some earlier discussoin -- I've been having trouble keeping up with the lists. But was there discussion of why this is a GUC? Why not just another parameter to EXPLAIN like the others? i.e. EXPLAIN (ANALYZE, BUFFERS, IOTIMING) -- greg ^ permalink raw reply [nested|flat] 12552+ messages in thread
* Re: Patch: add timing of buffer I/O requests 2012-03-21 15:01 Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-22 23:38 ` Re: Patch: add timing of buffer I/O requests Ants Aasma <[email protected]> 2012-03-27 18:58 ` Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-28 00:41 ` Re: Patch: add timing of buffer I/O requests Greg Stark <[email protected]> @ 2012-03-28 00:51 ` Robert Haas <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Robert Haas @ 2012-03-28 00:51 UTC (permalink / raw) To: Greg Stark <[email protected]>; +Cc: Ants Aasma <[email protected]>; Greg Smith <[email protected]>; pgsql-hackers On Tue, Mar 27, 2012 at 8:41 PM, Greg Stark <[email protected]> wrote: > On Tue, Mar 27, 2012 at 7:58 PM, Robert Haas <[email protected]> wrote: >> I've committed the core of this. I left out the stats collector >> stuff, because it's still per-table and I think perhaps we should back >> off to just per-database. I changed it so that it does not conflate >> wait time with I/O time. Maybe there should be a separate method of >> measuring wait time, but I don't think it's a good idea for the >> per-backend stats to measure a different thing than what gets reported >> up to the stats collector - we should have ONE definition of each >> counter. I also tweaked the EXPLAIN output format a bit, and the >> docs. > > Maybe I missed some earlier discussoin -- I've been having trouble > keeping up with the lists. > > But was there discussion of why this is a GUC? Why not just another > parameter to EXPLAIN like the others? > i.e. EXPLAIN (ANALYZE, BUFFERS, IOTIMING) Because you want to be able to expose the data even for queries that aren't explained. Right now, you can do that with pg_stat_statements; and the original patch also had per-table counters, but I didn't commit that part due to some concerns about stats bloat. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 544dd0d6e3d..13284f1d708 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index d89be1b9533..4fd10a5e7b3 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1164,9 +1164,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3694,6 +3691,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 4b37d92a49e..8d1e8a20998 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-= Content-Type: text/x-diff Content-Disposition: attachment; filename=v42-0007-Teach-snapshot-builder-to-skip-transactions-running-.patch ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 17118f2fe76..cb06c52c1d4 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1180,7 +1180,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index b1e05d56a1e..89aad2f7b77 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -714,6 +714,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -759,6 +762,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1275,7 +1279,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1454,6 +1459,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1544,7 +1555,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3017,6 +3028,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3027,7 +3039,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3078,6 +3091,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3835,7 +3849,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index fe419ef808e..9aaa8e95389 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3505,6 +3502,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 34209bd1393..ad5c1238e09 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1230,6 +1230,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 3426087b445..3778e9b7f07 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -65,6 +65,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -146,7 +147,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --=-=-=-- ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v43 6/7] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 98d38dc5229..1d28202de34 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1455,6 +1460,12 @@ index_create_copy(Relation heapRelation, Oid oldIndexId, Oid tablespaceOid, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1545,7 +1556,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3018,6 +3029,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3028,7 +3040,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3079,6 +3092,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3836,7 +3850,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 36950aea780..065316ee866 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1162,9 +1162,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3616,6 +3613,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 836eedb8ed5..1c4a887da13 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1231,6 +1231,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 144e241ebd3..6c7f0fee115 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -152,7 +153,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --nzintiedl6o4kcyp Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v43-0007-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
* [PATCH v44 09/10] Fix a few problems in index build progress reporting. @ 2026-03-16 12:53 Antonin Houska <[email protected]> 0 siblings, 0 replies; 12552+ messages in thread From: Antonin Houska @ 2026-03-16 12:53 UTC (permalink / raw) First, index_build() should not update the progress when being driven by REPACK, because the progress reporting infractructure cannot handle status of two commands at the same time. So far, REPACK with the CONCURRENTLY option neglected this problem altogether, but even the existing REPACK wasn't consistent enough: even if the 'progress' variable in repack_index() was false, it didn't pass the value to index_build(). Second, REPACK (CONCURRENTLY) should not set PROGRESS_REPACK_PHASE to PROGRESS_REPACK_PHASE_FINAL_CLEANUP in rebuild_relation() because it calls finish_heap_swap() anyway (via rebuild_relation_finish_concurrent()), which does the same thing. --- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/catalog/heap.c | 3 ++- src/backend/catalog/index.c | 22 ++++++++++++++++++---- src/backend/catalog/toasting.c | 3 ++- src/backend/commands/cluster.c | 7 ++++--- src/backend/commands/indexcmds.c | 1 + src/include/catalog/index.h | 4 +++- 7 files changed, 31 insertions(+), 11 deletions(-) diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 68a42de0889..1c8226c6d67 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -1181,7 +1181,7 @@ build_indices(void) heap = table_open(ILHead->il_heap, NoLock); ind = index_open(ILHead->il_ind, NoLock); - index_build(heap, ind, ILHead->il_info, false, false); + index_build(heap, ind, ILHead->il_info, false, false, false); index_close(ind, NoLock); table_close(heap, NoLock); diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 5748aa9a1a9..ae6b7cda3dd 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -3570,7 +3570,8 @@ RelationTruncateIndexes(Relation heapRelation) /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, currentIndex, indexInfo, true, false); + index_build(heapRelation, currentIndex, indexInfo, true, false, + true); /* We're done with this index */ index_close(currentIndex, NoLock); diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index de7182a85a9..cfd33f73f48 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -715,6 +715,9 @@ UpdateIndexRelation(Oid indexoid, * already exists. * INDEX_CREATE_PARTITIONED: * create a partitioned index (table must be partitioned) + * INDEX_CREATE_REPORT_PROGRESS: + * update the backend's progress information during index build. + * constr_flags: flags passed to index_constraint_create * (only if INDEX_CREATE_ADD_CONSTRAINT is set) * allow_system_table_mods: allow table to be a system catalog @@ -760,6 +763,7 @@ index_create(Relation heapRelation, bool invalid = (flags & INDEX_CREATE_INVALID) != 0; bool concurrent = (flags & INDEX_CREATE_CONCURRENT) != 0; bool partitioned = (flags & INDEX_CREATE_PARTITIONED) != 0; + bool progress = (flags & INDEX_CREATE_REPORT_PROGRESS) != 0; char relkind; TransactionId relfrozenxid; MultiXactId relminmxid; @@ -1276,7 +1280,8 @@ index_create(Relation heapRelation, } else { - index_build(heapRelation, indexRelation, indexInfo, false, true); + index_build(heapRelation, indexRelation, indexInfo, false, true, + progress); } /* @@ -1438,6 +1443,12 @@ index_create_copy(Relation heapRelation, bool concurrently, stattargets[i].isnull = isnull; } + /* + * Note: The current callers do not need INDEX_CREATE_REPORT_PROGRESS. If + * 'concurrently' is true, there is no build at all. Otherwise the index + * build is a sub-command of REPACK. The current infrastructure does not + * allow two commands to report their progress at the same time. + */ if (concurrently) flags = INDEX_CREATE_SKIP_BUILD | INDEX_CREATE_CONCURRENT; @@ -1528,7 +1539,7 @@ index_concurrently_build(Oid heapRelationId, indexInfo->ii_BrokenHotChain = false; /* Now build the index */ - index_build(heapRel, indexRelation, indexInfo, false, true); + index_build(heapRel, indexRelation, indexInfo, false, true, true); /* Roll back any GUC changes executed by index functions */ AtEOXact_GUC(false, save_nestlevel); @@ -3001,6 +3012,7 @@ index_update_stats(Relation rel, * * isreindex indicates we are recreating a previously-existing index. * parallel indicates if parallelism may be useful. + * progress indicates if the backend should update its progress info. * * Note: before Postgres 8.2, the passed-in heap and index Relations * were automatically closed by this routine. This is no longer the case. @@ -3011,7 +3023,8 @@ index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel) + bool parallel, + bool progress) { IndexBuildResult *stats; Oid save_userid; @@ -3062,6 +3075,7 @@ index_build(Relation heapRelation, RestrictSearchPath(); /* Set up initial progress report status */ + if (progress) { const int progress_index[] = { PROGRESS_CREATEIDX_PHASE, @@ -3819,7 +3833,7 @@ reindex_index(const ReindexStmt *stmt, Oid indexId, /* Initialize the index and rebuild */ /* Note: we do not need to re-establish pkey setting */ - index_build(heapRelation, iRel, indexInfo, true, true); + index_build(heapRelation, iRel, indexInfo, true, true, progress); /* Re-allow use of target index */ ResetReindexProcessing(); diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index 078a1cf5127..73c41360ae9 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -331,7 +331,8 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, BTREE_AM_OID, rel->rd_rel->reltablespace, collationIds, opclassIds, NULL, coloptions, NULL, (Datum) 0, - INDEX_CREATE_IS_PRIMARY, 0, true, true, NULL); + INDEX_CREATE_IS_PRIMARY | INDEX_CREATE_REPORT_PROGRESS, 0, + true, true, NULL); table_close(toast_rel, NoLock); diff --git a/src/backend/commands/cluster.c b/src/backend/commands/cluster.c index 2c3058ba10d..018e9f7b013 100644 --- a/src/backend/commands/cluster.c +++ b/src/backend/commands/cluster.c @@ -1153,9 +1153,6 @@ rebuild_relation(Relation OldHeap, Relation index, bool verbose, rebuild_relation_finish_concurrent(NewHeap, OldHeap, ident_idx, frozenXid, cutoffMulti); - - pgstat_progress_update_param(PROGRESS_REPACK_PHASE, - PROGRESS_REPACK_PHASE_FINAL_CLEANUP); } else { @@ -3582,6 +3579,10 @@ rebuild_relation_finish_concurrent(Relation NewHeap, Relation OldHeap, relpersistence = OldHeap->rd_rel->relpersistence; is_system_catalog = IsSystemRelation(OldHeap); + /* + * finish_heap_swap() sets the phase too, but here we need some swapping + * in advance. + */ pgstat_progress_update_param(PROGRESS_REPACK_PHASE, PROGRESS_REPACK_PHASE_SWAP_REL_FILES); diff --git a/src/backend/commands/indexcmds.c b/src/backend/commands/indexcmds.c index 83edab38760..c9e22e95515 100644 --- a/src/backend/commands/indexcmds.c +++ b/src/backend/commands/indexcmds.c @@ -1233,6 +1233,7 @@ DefineIndex(ParseState *pstate, flags |= INDEX_CREATE_PARTITIONED; if (stmt->primary) flags |= INDEX_CREATE_IS_PRIMARY; + flags |= INDEX_CREATE_REPORT_PROGRESS; /* * If the table is partitioned, and recursion was declined but partitions diff --git a/src/include/catalog/index.h b/src/include/catalog/index.h index 56a064ef444..a03238304a0 100644 --- a/src/include/catalog/index.h +++ b/src/include/catalog/index.h @@ -71,6 +71,7 @@ extern void index_check_primary_key(Relation heapRel, #define INDEX_CREATE_IF_NOT_EXISTS (1 << 4) #define INDEX_CREATE_PARTITIONED (1 << 5) #define INDEX_CREATE_INVALID (1 << 6) +#define INDEX_CREATE_REPORT_PROGRESS (1 << 7) extern Oid index_create(Relation heapRelation, const char *indexRelationName, @@ -148,7 +149,8 @@ extern void index_build(Relation heapRelation, Relation indexRelation, IndexInfo *indexInfo, bool isreindex, - bool parallel); + bool parallel, + bool progress); extern void validate_index(Oid heapId, Oid indexId, Snapshot snapshot); -- 2.47.3 --gwom7bl7ogtszo4k Content-Type: text/x-diff; charset=utf-8 Content-Disposition: attachment; filename="v44-0010-Teach-snapshot-builder-to-skip-transactions-runn.patch" ^ permalink raw reply [nested|flat] 12552+ messages in thread
end of thread, other threads:[~2026-03-16 12:53 UTC | newest] Thread overview: 12552+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2012-03-21 15:01 Re: Patch: add timing of buffer I/O requests Robert Haas <[email protected]> 2012-03-22 23:38 ` Ants Aasma <[email protected]> 2012-03-27 18:58 ` Robert Haas <[email protected]> 2012-03-27 19:22 ` Ants Aasma <[email protected]> 2012-03-28 00:30 ` Robert Haas <[email protected]> 2012-03-27 19:23 ` Robert Haas <[email protected]> 2012-03-28 00:41 ` Greg Stark <[email protected]> 2012-03-28 00:51 ` Robert Haas <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v43 6/7] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]> 2026-03-16 12:53 [PATCH v44 09/10] Fix a few problems in index build progress reporting. Antonin Houska <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox